Technical notes
Survey Management
The BRFSS is a cooperative effort of the Centers for Disease Control and Prevention and participating states. The CDC develops the core questionnaire and provides training, technical assistance, standardized data analyses and funding. The Pennsylvania Department of Health develops supplemental questions (or modules) and responds to requests for data. In 2020, Pennsylvania was divided into eight strata. These strata consisted of eight regional areas: six Pennsylvania health districts and Allegheny and Philadelphia counties. Sampling and interviewing in 2020 were done by the University of Pittsburgh Evaluation Institute.
Questionnaire
The survey questionnaire for the statewide survey of Pennsylvania consists of a standardized core, state-selected modules and state-added questions. The CDC developed the core questionnaire with recommendations from all participating jurisdictions. Most of the core questions had been used during the 2019 BRFSS survey. All items new to the 2020 survey were field-tested.
Questions of interest to Pennsylvania were added as the state supplement to the core questionnaire. State-added modules and questions in 2020 concerned pre-diabetes, oral health, industry and occupation, occupational health, e-cigarettes, caregiver, childhood asthma prevalence, health literacy, environmental health, gambling, COVID-19, and opioid use.
Response Rates
Interviews were conducted in the evenings and on weekends to reach people when they were more likely to be at home, as well as during the day. For landline sample, at least 6 calls were placed at different times of the day and night on different days of the week before any sample number was classified as "no answer."
Interviewers who were experienced in converting refusals to completed interviews recontacted people who refused to participate in the survey.
For cell phone samples, if a number has not been reached within the first six attempts, the record will receive a final disposition code on the seventh attempt. However, if any contact has been made within the first six attempts, the number will receive additional attempts, up to a total of 12.
The final dispositions of the landline and cell phone samples, as recorded by the data collection firm, are shown in the tables in the dispositions section of this report. These dispositions allow calculation of the Council of American Survey Research Organizations (CASRO) response rates. CASRO response rates may be thought of as the percentage of eligible telephone numbers that yielded an interview, adjusted in a standard way for the large quantity of telephone numbers of unknown eligibility. The CASRO response rates for the landline and cell phone samples for the 2020 survey are 43.8 and 38.7, respectively.
Determining Accuracy of the Estimates and Significance Using Confidence Intervals
Tables included in this report show the 95% confidence intervals associated with all reported percentages. They appear in the table columns labeled (CI).
Confidence intervals are a way to measure sampling error and define the range of values where percentages estimated by multiple samples of the same population would be found (95 percent of the time). The size of the confidence interval is directly related to the probability of selection and characteristics of the people surveyed within the universe being sampled. Percentages for two different subgroups of the population are significantly different if their confidence intervals or ranges do not overlap.
Confidence intervals were calculated using SUDAAN, a software package developed by the Research Triangle Institute that properly estimates sample variances for complex sample designs.
Percentages were not calculated and shown for subgroups of the population when their sample size was less than 50. The method used to determine the reliability of percentages calculated from sample sizes of 50 or more consisted of a comparison of the relative standard error of the calculated percentage with the relative standard error of the same percentage outcome for a simple random sample. If the relative standard error for the percentage being tested was smaller than the relative error of the same percentage outcome for the simple random sample, then the calculated percentage was considered reliable.
Weighting and Sample Frame Changes in 2011
In 2011, the BRFSS added a sample of cell phone numbers to the sample of landline telephones used in previous years. This was necessary due to the increasing proportion of adults in the United States and in Pennsylvania who live in cell phone-only households. These people tend to be younger, more mobile, and to belong to racial and ethnic minority groups. The loss of this group to the survey population introduced bias to the estimates.
A new weighting methodology called "iterative proportional fitting," or "raking," which is explained in the "Data Adjustment" section, includes the telephone source (landline or cell phone) in the weighting methodology. It also allows the BRFSS to adjust the sample interviews to match the characteristics of the population not only on age, sex and race, but also on marital status, education, and ownership or rental of the home.
These changes amount to a major re-working of the BRFSS survey and will shift estimates and trend lines in ways that are not related to changes in the actual population. It is advisable to re-benchmark any trend lines and measures, using 2011 as a new starting point.
The new "raking" weighting method was the primary means of combining the samples of landline telephones and cell phones so that they would yield valid estimates of the entire adult household population. However, changes have been made each year after 2011 to deal with the possibility that the same household may appear in both samples. The 2011 survey excluded households from the cell phone sample if they had a landline telephone. The 2012 survey excluded households from the cell sample if fewer than 90 percent of their calls were received on a cell phone. The 2013 survey did the same but reweighted interviews from the landline sample if the household reported cell phone usage of 90 percent or more and reweighted interviews from the cell phone sample that also had a landline telephone in use (called "cell-mostly"). Since 2014, the survey included all households reached from either cell or landline samples and resolved the effects of possible duplication with weighting.
Data Adjustment
Before 2011, BRFSS weights were based on several design factors, such as the number of adults and the number of telephones in the household, as well as differential sampling of households that are more and less easily reached. After design factors were taken care of, an additional factor was applied that compensated for shortfalls due to non-response and undercoverage.
This was called the post-stratification adjustment. Usually, the Pennsylvania survey has been post-stratified by two sexes and at least six age groups, involving six age-by-sex categories, for a total of 12 cells.
Beginning with the 2011 sample, the weighting process became much more complex. After weights for design factors were applied, the sample was adjusted, using the "raking" method to agree proportionally with more external references (called margins).
The adjustments are applied to the sample sequentially, with the objective of bringing the total of each cell of each margin into the same proportion as the target population. The proportion for the cells of the first margin (age x gender) are processed and the sample is weighted, then another margin (race/ethnicity) is processed and the sample is weighted again. When the sample has been weighted for all the margins once, the process repeats, reweighting the sample. This continues until no cell in any margin differs from the target proportions by more than 0.025 percent (i.e., convergence) or until the margins have been processed 75 times.
For states that use regional weighting, as Pennsylvania does, there are 12 raking control margins as shown below.
| 1. | Age group by gender |
| 2. | Detailed race/ethnicity |
| 3. | Education |
| 4. | Marital status |
| 5. | Tenure (rent of own home) |
| 6. | Gender by race/ethnicity |
| 7. | Age group by race/ethnicity |
| 8. | Phone usage groups |
| 9. | Region |
| 10. | Region by age group |
| 11. | Region by gender |
| 12. | Region by race/ethnicity |
Beginning with the 2013 survey, CDC added four additional geographic margins. The geographies (Allegheny, Philadelphia and the rest of the state) are similar to stratification that Pennsylvania already uses and may be more important in other states.
| 13. | County |
| 14. | County by race/ethnicity |
| 15. | County by age group |
| 16. | County by gender |
The sample design in Pennsylvania includes the six districts and Allegheny and Philadelphia counties, making eight regions. For those states that do not use regional weighting, only the first eight control variables are used in the raking. The population control totals for age group, gender, race/ethnicity and region and the control percentages for education, marital status and tenure are obtained from the Nielsen Company, Inc. and the American Community Survey. The estimate that 49.8 percent of the 2019 Pennsylvania adult population are cell phone-only users is derived from the National Health Interview Survey (NHIS) [See Blumberg, et al.].
All percentages reported here were calculated with weighted data in order to be representative of the adult population of Pennsylvania. It should be noted that the percentages might not add to 100, due to rounding. When calculating the percentages of prevalence for each health topic in this report, responses of "Don't know/Not sure" and "Refused" were removed from the denominators.
Reports based on BRFSS data can come from a wide variety of sources. The record level data from every state can be downloaded from the BRFSS website (http://www.cdc.gov/brfss/), with certain fields suppressed to protect the confidentiality of respondents. Other sources may include or exclude data from the denominators used to calculate percentages. For example, the exclusion or inclusion of "Don't know/Not sure" or "Refused" responses could potentially affect the final response percentage calculation.
If you have any questions about these differences, please contact the Bureau of Informatics and Information Technology by email at ra-dhicontactus@pa.gov.